Apparatus and method for registration of points of a data field with respective points of an optical image

ABSTRACT

An apparatus and method enables a precise superimposition of an optical representation with a data field to support the navigation during endoscopic operations. An optical representation is acquired by an endoscope with a camera and displayed on a screen. A sensor is attached to the endoscope or camera to continuously detect its spacial position. A spacial data field is also acquired, from such techniques as X-ray tomography, NMR tomography or ultrasound, and allocated to a body in a certain position. A sensor which is adapted to be attached to the body compensates for any movement of the body. A computer is provided for registering selected points of the data field with respective points of the optical representation by displacement certain points of the data field while superimposed on the optical representation.

BACKGROUND

1. Field of the Invention

The invention relates to a method for representing the interior ofbodies with the following steps:

Providing an optical imaging system consisting of a camera and amonitor;

allocation of a spatial data field to the body disposed in a certainposition;

continuous detection of the spatial position of the camera;

continued calculation of a representation of the data field whichcorresponds to the current angle of view of the camera;

simultaneous or alternative representation of the optical image and thedata field on the monitor.

2. The Background Art

Endoscopes are used with increasing frequency in operations so as toreduce the stress for the patients. During this process the endoscopicimage is represented on a video monitor. This means a substantial changein operating technique for the doctor.

In common operations the operating field is freely accessible to theeye, and there is a natural coordination of the hand movements. This isno longer the case in operations by means of an endoscope. There is noconnection between the orientation of the endoscope and the direction ofview of the user, i.e., the operating surgeon. As a result of this themovement of surgical instruments relative to the endoscope becomesdependent on the surgeon's faculty of three-dimensional visualization.The second disadvantage is the lack of spatial feeling, as usually onlyone endoscopic lens is used. For each operation it is generallynecessary to have knowledge of organ and tumor borders and theanatomical situation. An overview over the operating field facilitatesorientation.

The third aspect is planning the operation. In a freely accessibleoperating field there is a clear sequence of operating steps. Thesurgical instruments can be used intuitively. The operation by means ofendoscopes places higher requirements. Even the positioning of thesurgical instruments relative to the operating field requires planning.

In the field of stereotactic surgery there are methods which can be usedin principle for endoscopical surgery too.

From DE-A 37 17 871 it is known to mix in data such as computertomography (CT) representations into the operating microscope in orderto obtain help in the navigation of surgical instruments. Therepresented CT-layers correspond to the plane to which the microscope isfocussed. During a movement of the instrument the respective layers aredisplayed dynamically on the computer screen. The surgeon is to besupported in this way in the positioning of an instrument relative to ananatomical structure. In order to bring the microscopic image with theCT-representation into alignment, it is necessary that certain pointswhich are marked by means of a laser beam are aimed at with themicroscope and that thereafter the microscope is focused therethrough.

U.S. Pat. No. 4,722,056 describes a method in which a tomography imageis overlapped with the focal plane of an operating microscope. Therepresentation of a CT-layer is adjusted to the optical representation.

DE-A 41 34 481 relates to a microscope for stereotactic microsurgery. Alaser locating system is used for determining the position of themicroscope relative to the patient. The function is similar to that ofthe microscope which is described in U.S. Pat. No. 4,722,056.

In EP-A 488 987 of the applicant a method is described for overlappingdata and optical representations. With this method it is possible, forexample, to mix in axes of extremities into an optical representation inrealtime.

In the field of endoscopic surgery complete CT-series of findings arerarely available. Moreover, the spatial reproduceability of the positionof anatomical structures is limited primarily to the skull. In theabdominal region the intraoperative condition is not deduceable from apreoperative CT without any special measures. Furthermore, computertomography is a relatively complex method which is not always readilyavailable or cannot be used.

These known methods assume that the position of the patient prior to theoperation can be determined definitely relative to a spatial coordinatesystem with all three axes of freedom. This can be made, for example, byfocussing marking points with an operating microscope. After thedetermination it is necessary that the patient remain rigidly fixed,i.e., the patient must be strapped in a fixed manner to the operatingtable. The position of the microscope is detected in this known methodvia the rod structure or via position sensors, so that aCT-representation or the like can be brought in relationship to theimage plane, which allows a superimposition of this representation withthe optical image.

These methods are used primarily in the field of stereotactic surgery. Asurgical instrument is to be guided towards a tumor, for example. Theposition of the tumor is reconstructed from CT-findings. No change inthe position of the patient per se or the position of the operatingfield within the patient may occur after acquiring the position of thepatient, i.e., particularly during the operation.

However, a completely rigid fixation of a patient is not alwayspossible. Moreover, additional difficulties occur particularly duringendoscopic operations. The endoscope is moved to the target zone throughopen cavities in the body. These are generally relatively flexible andtherefore rarely correlate with CT-findings. Moreover, tissue mayfrequently be displaced considerably during an operation, e.g. byremoving parts of tissue, suction of liquid, etc. As a result of thisthe representation of the data field correlates less and less with theoptical image and the information provided becomes increasinglyworthless.

Moreover, it is to be observed that owing to the limited precision ofposition sensors an optimal determination of position is always onlypossible for a specific spatial volume. Marking points which undercertain circumstances may be relatively far away from the target zone asis generally the case in endoscopic methods are not optimal with respectto the achievable precision. Finally, a certain temporal drift occurs inposition sensors so that unavoidable deviations will occur during longeroperations.

BRIEF SUMMARY AND OBJECTS OF THE INVENTION

It is the object of the present invention to avoid said disadvantagesand to provide a method which enables a precise superimposition of theoptical representation with a data field, e.g., a CT-representation,during the use of an endoscope too.

It is a further object of the invention to provide a method forsupporting the navigation during endoscopic operations which can be usedwithout the presence of representations from computer tomography.

This object is achieved in that an endoscope is series-connected to thecamera, that a calibration is carried out repeatedly which consists ofbringing in conformity one or several characteristic points of the datafield with the pertinent optical representation on the screen by anentry process of the user.

The substantial element in the invention is that marking points, as inthe state of the art, are only used for "rough navigation", if required.This allows approximate aiming at the target region. In the actualnavigation the user is free in the choice of the points used forre-calibration. The re-calibration can therefore always be made in theregion of particular interest, thus maximizing the precision in thisregion. In contrast to the methods of the state of the art, in which thework practically proceeds from the outside to the inside, the process ofthe present invention can be designated as a process from the inside tothe outside.

When using novel 3D-sensors sensing chips, which have a size of approx.1 mm² only, it is possible to secure a plurality of such sensors to thepatient in order to create a local reference coordinate system. Approx.100 of such sensors can be applied in the liver region. There-calibration in accordance with the invention is bringing an opticalimage into registration with the determination of the position byreference sensors.

A representation gained from an imaging method such as X-ray tomography,NMR tomography, an ultrasonic representation or the like can be used asa data field. In order to obtain representations which are moredescriptive than common sectional representations it is possible torework the CT-findings in order to maintain characteristic points orlines which are particularly suitable for comparisons or for renewedfinding. Such a method may be as has been described, for example, by N.Ayache et al.: "Evaluating 3D Registration of CT-Scan Images Using CrestLines", in: SPIE Vol. 2035 Mathematical in Medical Imaging II (1993), p.60.

It is particularly advantageous when a three-dimensional reconstructionis used as a data field which is obtained from previously made videorecordings. In this way it is possible to provide a navigational aidwithin the scope of the invention without allowing the necessity toarise that a CT-representation has to be made. Either prior to theoperation or in an early stage of the operation a local3D-reconstruction of the operating field is made. This allows a preciseplanning of the operation. After carrying out changes in the operatingfield, e.g., by excision of parts of tissue, tumors, etc., therepresentation of the condition existing beforehand can be overlappedwith the current condition.

It is possible that the three-dimensional reconstruction is obtainedfrom a single video recording to which a distance measurement isallocated, e.g., via ultrasonic sound.

On the other hand, the three-dimensional reconstruction can be obtainedfrom several video recordings by stereometric analysis. Such astereometric analysis is known, for example, from P. Haigron, : "3DSurface Reconstruction Using Strongly Distorted Stereo Images", in:IEEE, Proceedings of the Annual Conference on Engineering in Medicineand Biology (1991), IEEE cat. n. 91CH3068-4, p. 1050f. This paperdescribes the reconstruction of the surface of the femur in the kneearea by distorted endoscopic images. The spatial reconstruction fromsingle images is described in a general way by Fua. P.: "CombiningStereo, Shading and Geometric Constraints for Surface Reconstructionfrom Multiple Views", in SPIE Vol. 2031 Geometric Methods in ComputerVision II (1993), p. 112ff.

It is advantageous if the displacement of the points is made by means ofa computer mouse. It is not always possible to target specific pointswhich are to be used for recalibration with the endoscope which isinserted into a body cavity so that they come to lie precisely in thegraticule. It is far easier to bring the points of interest only intothe field of vision of the endoscope and then to fix the image, i.e., tofreeze it, and then to carry out the matching. In this process it isalso possible to process several points simultaneously.

It may further be provided that the endoscope is used for examining apatient to which a position sensor is attached so as to compensate anychanges in the position of the patient. This measure allows moving thepatient also during the work with the endoscope. When the coordinatesystem of the target zone is not displaced relative to the coordinatesystem of the whole patient, a re-calibration is not necessary.

The invention further relates to an apparatus for carrying out the abovemethod. The apparatus may include the following elements:

a camera with an endoscope attached thereto;

a position sensor attached to the camera or the endoscope;

a monitor for displaying the optical image recorded by the cameratogether with a data field;

a computer with a memory for the data field and means for detecting theposition of the position sensor.

The apparatus is characterized in that means are provided which allowthe user to bring into conformity points of the data field withrespective points of the optical image and thus to improve theconformity between the other representation of the data field with theoptical image. These means may include, for example, a mouse as is usedfrequently as an input medium for computers and of respective algorithmsfor the readjustment of the coordinate transformations so as to obtainfrom the entry of the user a better "fit" between the opticalrepresentation and the representation of the data field.

The invention is now explained in closer detail by reference to anembodiment shown in the drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows schematically the representation of an endoscopic image onthe monitor prior to calibration;

FIG. 2 shows a respective representation after the calibration;

FIG. 3 shows a test image for correcting the distortion of theendoscopic image;

FIG. 4 shows the representation of the test image which is distorted bythe endoscope;

FIG. 5 shows schematically the endoscope attached to a video camera;

FIG. 6 shows schematically a reference object for determining thespatial position of the image plane;

FIG. 7 shows a screen representation for calibrating the image plane ofthe camera;

FIG. 8 shows schematically the different coordinate systems and theirrelationships, including the hardware involved.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Image section 1 shows an instrument 2 with a tip 2a. The point 3represents the cross-faded calculated position of the tip, i.e., the"virtual" image of the tip. In FIG. 1 the real image 2a and the virtualimage 3 fall apart. By making respective adjustments it is possible tobring the images into conformity, as is shown in FIG. 2. The calibrationis thus completed.

In the same way it is possible to carry out the calibration with acharacteristic point 4a of a represented object 4. In FIG. 1 the opticalrepresentation 4a and the virtual image 5 fall apart. After thecalibration this is no longer the case.

The test image shown in FIG. 3 includes points 10 arranged evenly in asquare pattern. The points 10 may be represented by dots of suitablesize or may be connected by lines. This image is distorted by theoptical system of the endoscope, as is shown in FIG. 4. A respectiveoverlap of other representations is thus provided with errors, which arethe greater the farther the respective detail is disposed outside of thecenter 11. To improve the conformity the distorted position of theindividual measured points 10 is determined by a respective imageprocessing program. As the true position is known with the exception ofa scaling factor determinable by the user, a distortion function can becalculated for the entire image plane. With mathematical methods whichare known to the man skilled in the art it is possible to calculate acorrection function by inverting this function, which removes thedistortion. It is clear that this process must be carried out for eachendoscope, as endoscopes of the same type can well be provided withdifferent distortions. A pattern recognition method may be used toevaluate and correct distortions.

FIG. 5 shows a video camera 20 with an endoscope 21 attached thereto.The respective position of the camera 20, and thus of endoscope 21, isdetermined via a 3D-sensor 22. The image plane of the camera isdesignated with 23.

FIG. 6 shows schematically a fixed reference object 30 in the form of acube for determining the position of the image plane of the camera. Thecoordinates x, y, z of the corner points a, b of the cube in a spatialcoordinate system are known. Either the coordinates of a cube 30 whichis immobile in space is acquired with a 3D-digitizer or, as is shown inFIG. 6, a position sensor 30a is attached to cube 30, by means of whichsaid cube 30 can be freely movable in space also during thedetermination of the position. In a symbolic drawing of this cube 30which is shown on the screen the user must bring the corner points to amatching position with the frozen image by displacement. From thisinformation the computer is enabled to calculate the coordinates of theimage plane present at the time of freezing the image with the help of adirect linear transformation. With the help of the 3D-sensor 22 attachedto camera 20 it is also possible to calculate the position of the imageplane which might have possibly changed in the meantime.

FIG. 7 shows an image section which can be used in calibrating theposition of the image plane. Representations of the reference object,namely cube 30, recorded from different positions are shown in threesectors 35a, 35b and 35c. In this way it is possible to carry out aplurality of calibrating measurements in a single image representationin order to maximize the precision.

As an alternative to an actually existing reference object it is alsopossible to use a "virtual reference object" for calibrating theendoscope. A camera 20 is aimed at the tip of a 3D-stylus. A 3D-stylusis a device of the size of a ballpoint pen which can issue data on thespatial position of its tip at any time via built-in magnetic coils. Thecalibration of the endoscope is made in such a way that the camera isaimed at the 3D-stylus. The image is then fixed, i.e., it is frozen, anda cursor disposed on the screen is moved with a mouse or joystick to therepresentation of the tip. This process is carried out at least sixtimes. In this way a precise calibration of the endoscope is possible.

The precision of the overlapping can be made in a very simple and clearmanner in that the 3D-stylus is brought into the image. The opticalrepresentation of the tip and the symbolic display gained from thecoordinates have to be precisely above one another in the case of anoptimal adjustment. Any distance shows an imprecise adjustment of thecoordinate systems.

FIG. 8 schematically shows the spatial coordinate system 40. It isrepresented by way of example by the digitizer stylus 50, with which thecoordinates of every spatial point can be determined by scanning. Thecoordinate system 41 is the one of endoscope 61 or the camera 51 fixedlyattached thereto. The current position is detected via the fixedlyattached position sensor 71. The calibration of the image plane is madeonce via a reference object or a virtual reference object, as isdescribed above. In this way the relationship 81 between coordinatesystems 40 and 41 is determined for the duration of the endoscopicexamination.

A position sensor 72 may be attached to patient 52, who may lie on anoperating table 62. The relationship 82 to the spatial coordinate systemis determined by a one-off adjustment, and thus relationship 90 isdetermined too.

The target zone is indicated with reference numeral 53. Its coordinatesystem, which is also the one of the data structure, can be detectedroughly at first by a setup on the basis of external conditions. Adirect tracing is not possible, which is why the relationship 83 isshown in a broken line. The method in accordance with the invention,however, allows establishing the relationship 92 to the camera 51 or theendoscope 61, by means of which relationship 91 is also determined. Whenrelationship 91 changes, e.g., after the removal of parts of tissue, itis necessary to perform a re-calibration in accordance with the methodof the invention.

The required calculations are made in computer 24 and displayed onmonitor 25. The mouse 26 is used for carrying out the recalibration.

The optical image and the date image may each be displayed separately.Alternatively, a user may selectively view the optical image and thedata image as superimposed images displayed on the monitor.

I claim:
 1. A method for representing the interior of bodies, the methodcomprising:providing an optical imaging system comprising a camera and amonitor, wherein an endoscope is connected in series with the camera tocreate an optical image; attaching a position sensor to a body tocompensate for changes in position of the body; allocating a spatialdata field to the body disposed in a certain position; identifyingpoints on the body and identifying characteristic points, correspondingthereto, in the data field; calibrating the optical imaging system byregistering the characteristic points with the points on the body;detecting continuously the spatial position of the camera; calculating arepresentation of the data field corresponding to a current angle ofview of the camera; and selectively representing the optical image, thedata field, and both the optical image and the data field superimposed,on the monitor.
 2. A method as claimed in claim 1, wherein the datafield is provided by using an imaging method selected from X-Raytomography, NMR tomography, and ultrasonic imaging.
 3. A method asclaimed in claim 1, further comprising obtaining a video recording ofthe image and providing a three-dimensional reconstruction thereof.
 4. Amethod of claim 3, wherein the three-dimensional reconstruction isgained from a single video recording to which a distance measurement isallocated.
 5. A method of claim 3, wherein the three-dimensionalreconstruction is gained from several video recordings by stereometricanalysis.
 6. A method of claims 1 to 5, further comprising an entryprocess comprising freezing the optical image superimposed on the datafield at a time determined by a user, and wherein individual points ofthe data field are displaceable on the monitor in said frozen opticalimage.
 7. A method as claimed in claim 6, wherein the displacement ofthe points is made by means of a computer mouse.
 8. A method of claim 1,comprising the step of correcting a distortion of the optical imagingsystem by aiming the endoscope towards a test image comprising regularlyarranged symbols, evaluating the symbols using a pattern recognitionmethod, and calculating a correction function for the optical system. 9.An apparatus for imaging the interior of bodies; the apparatuscomprising:a camera having an endoscope operably attached thereto; afirst position sensor attached to the camera or the endoscope to providea frame of reference with respect to the camera; a second positionsensor adapted to be attached to a body to compensate for changes inposition of the body; a monitor for displaying an optical image recordedby the camera together with a data field representing a model to besuperimposed on the optical image; a computer comprising a memory devicefor storing the data field representing the model; means for detectingthe position of the first position sensor; and means for registeringselected points of the data field representing points of the model withrespective points of the optical image by displacement of a second imagecorresponding to at least one point of the data field displayed on themonitor to be superimposed on the optical image.